Source type definition configuration using a graphical user interface

ABSTRACT

A data intake and query system provides interfaces that enable users to configure source type definitions used by the system. A data intake and query system generally refers to a system for collecting and analyzing data including machine-generated data. Such a system may be configured to consume many different types of machine data generated by any number of different data sources including various servers, network devices, applications, etc. At a high level, a source type definition comprises one or more properties that define how various components of a data intake and query system collect, index, store, search and otherwise interact with particular types of data consumed by the system. The interfaces provided by the system generally comprise one or more interface components for configuring various attributes of a source type definition.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to configuration of a dataintake and query system and, more particularly, to customizingconfiguration parameters related to processing types of data that may beprovided as input to the system.

BACKGROUND

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

Modern data centers and other computing environments often compriseanywhere from a few devices to thousands of computing devices thatprocess various types of data, service requests from an even largernumbers of remote clients, and perform many other computing functions.During operation, many of these devices may include components thatproduce significant volumes of machine-generated data. For example, manyof the devices may include components that produce various types of logfiles, output files, network data, etc.

Analysis of data generated by such computing devices may yield valuableinsight into both the overall operation of such computing environmentsand individual components thereof. However, the unstructured nature ofmuch of this data presents a number of challenges to analysis in partbecause of the difficulty of applying semantic meaning to unstructureddata. Furthermore, the data generated by the computing devices may varywidely both in the type and format of the data. As the number computingdevices that generate various forms of machine data continues to grow,processing and analyzing large volumes of such machine data in anintelligent manner and effectively presenting the results of suchanalysis remains a priority.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates a block diagram of an example data intake and querysystem in which an embodiment may be implemented;

FIG. 2 is a flow diagram that illustrates how indexers process, index,and store data received from forwarders in accordance with the disclosedembodiments;

FIG. 3 is a flow diagram that illustrates how a search head and indexersperform a search query in accordance with the disclosed embodiments;

FIG. 4 illustrates a block diagram of a system for processing searchrequests that uses extraction rules for field values in accordance withthe disclosed embodiments;

FIG. 5 illustrates an example search query received from a client andexecuted by search peers in accordance with the disclosed embodiments;

FIG. 6A illustrates a search screen in accordance with the disclosedembodiments;

FIG. 6B illustrates a data summary dialog that enables a user to selectvarious data sources in accordance with the disclosed embodiments;

FIG. 7A illustrates a key indicators view in accordance with thedisclosed embodiments;

FIG. 7B illustrates an incident review dashboard in accordance with thedisclosed embodiments;

FIG. 7C illustrates a proactive monitoring tree in accordance with thedisclosed embodiments;

FIG. 7D illustrates a screen displaying both log data and performancedata in accordance with the disclosed embodiments;

FIG. 8 illustrates a block diagram of an example cloud-based data intakeand query system in which an embodiment may be implemented;

FIG. 9 is a flow diagram that illustrates an example process forproviding interfaces enabling configuration of source type definitionsand synchronizing source type definitions across multiple systemcomponents, according to one or more aspects of the present disclosure;

FIG. 10 depicts an example graphical user interface displaying a list ofexisting source type definitions of a data intake and query system;

FIG. 11 depicts a portion of an example graphical user interfacedisplaying interface components for configuring a source typedefinition;

FIG. 12 depicts a portion of an example graphical user interface forpreviewing a source type configuration;

FIG. 13 illustrates a computer system upon which an embodiment may beimplemented.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

Embodiments are described herein according to the following outline:

1.0. General Overview

2.0. Operating Environment

-   -   2.1. System Overview    -   2.2. Data Server System    -   2.3. Source Types Overview    -   2.4. Data Ingestion        -   2.4.1. Input        -   2.4.2. Parsing        -   2.4.3. Indexing    -   2.5. Query Processing    -   2.6. Field Extraction    -   2.7. Example Search Screen    -   2.8. Acceleration Techniques        -   2.8.1. Map-Reduce Technique        -   2.8.2. Keyword Index        -   2.8.3. High Performance Analytics Store        -   2.8.4. Accelerating Report Generation    -   2.9. Security Features    -   2.10. Data Center Monitoring    -   2.11. Cloud-Based System Overview

3.0 Source Type Management

-   -   3.1. Configuring Source Types    -   3.2. Synchronizing Source Type Definitions

4.0. Implementation Mechanisms—Hardware Overview

5.0. Example Embodiments

6.0. Extensions and Alternatives

1.0. General Overview

This overview presents a basic description of some aspects of a possibleembodiment of the present invention. It should be noted that thisoverview is not an extensive or exhaustive summary of aspects of thepossible embodiment. Moreover, it should be noted that this overview isnot intended to be understood as identifying any particularlysignificant aspects or elements of the possible embodiment, nor asdelineating any scope of the possible embodiment in particular, nor theinvention in general. This overview merely presents some concepts thatrelate to the example possible embodiment in a condensed and simplifiedformat, and should be understood as merely a conceptual prelude to amore detailed description of example possible embodiments that followsbelow.

The present disclosure is directed to systems and techniques forenabling users to configure source type definitions used by a dataintake and query system. An example data intake and query system asdescribed herein generally refers to a system for collecting andanalyzing data including machine-generated data. Such a system may beconfigured to consume many different types of machine data generated byany number of different data sources, including machine data generatedby various servers, network devices, applications, etc. At a high level,a source type definition comprises one or more attributes that definehow various components of a data intake and query system collect, index,store, search and otherwise interact with particular types of dataconsumed by the system.

According to an embodiment, a data intake and query system provides oneor more user interfaces that enable users to create, modify, and deletesource type definitions. For example, a user may desire to create ormodify one or more source type definitions to customize how the systeminteracts with certain types of data of interest to the user (e.g.,particular types of log files or other data the user desires toanalyze). User interfaces for creating and modifying source typedefinitions generally may include one or more interface components thatenable users to specify values or rules corresponding to variousattributes of a source type definition. In general, one or morecomponents of a data intake and query system may generate the userinterfaces, respond to user input, and create and store source typedefinitions in one or more configuration files locally stored on asystem component.

As indicated above, a source type definition may be referenced at manydifferent data processing stages (e.g., data collection, indexing, andsearching) and by various different components of a data intake andquery system. Accordingly, in one embodiment, a data intake and querysystem is configured to synchronize source type definitions acrossmultiple components of such a system so that source type configurationsmade on one system component (e.g., a component that indexes the data)are accessible to other system components (e.g., other components thatcollect the data or perform searches on the data). By synchronizing asource type definition across multiple components of a data intake andquery system, input data received by the system for analysis may beprocessed consistently across different processing stages.

Other embodiments include, without limitation, a non-transitorycomputer-readable medium that includes processor-executable instructionsthat enable a processing unit to implement one or more aspects of thedisclosed methods as well as a system configured to implement one ormore aspects of the disclosed methods.

2.0. Operating Environment

2.1. System Overview

Modern data centers and other computing environments can compriseanywhere from a few host computer systems to thousands of systemsconfigured to process data, service requests from remote clients, andperform numerous other computational tasks. During operation, variouscomponents within these computing environments often generatesignificant volumes of machine-generated data. In general,machine-generated data can include performance data, diagnosticinformation, and many other types of data that can be analyzed todiagnose performance problems, monitor user interactions, and to deriveother insights.

A number of tools are available to analyze machine data. In order toreduce the size of the potentially vast amount of machine data that maybe generated, many of these tools typically pre-process the data basedon anticipated data-analysis needs. For example, pre-specified dataitems may be extracted from the machine data and stored in a database tofacilitate efficient retrieval and analysis of those data items atsearch time. However, the rest of the machine data typically is notsaved and is discarded during pre-processing. As storage capacitybecomes progressively cheaper and more plentiful, there are fewerincentives to discard these portions of machine data and many reasons tokeep it.

This plentiful storage capacity is presently making it feasible to storemassive quantities of minimally processed machine data for laterretrieval and analysis. In general, storing minimally processed machinedata and performing analysis operations at search time can providegreater flexibility because it enables an analyst to search all of themachine data, instead of searching only a pre-specified set of dataitems. This may, for example, enable an analyst to investigate differentaspects of the machine data that previously were unavailable foranalysis.

However, analyzing and searching massive quantities of machine datapresents a number of challenges. For example, a data center may generatemany different types of machine data (e.g., system logs, network packetdata, sensor data, etc.) from thousands of different components, whichcan collectively can be very time-consuming to analyze. Also, theunstructured nature of much of this machine data can pose additionalchallenges because of the difficulty of applying semantic meaning tounstructured data, and the difficulty of indexing and queryingunstructured data using traditional database systems.

These challenges can be addressed by using an event-based data intakeand query system, such as the SPLUNK® ENTERPRISE system produced bySplunk Inc. of San Francisco, Calif. The SPLUNK® ENTERPRISE system isthe leading platform for providing real-time operational intelligencethat enables organizations to collect, index, and searchmachine-generated data from various websites, applications, servers,networks, and mobile devices that power their businesses. The SPLUNK®ENTERPRISE system is particularly useful for analyzing unstructureddata, which is commonly found in system log files, network data, andother data input sources. Although many of the techniques describedherein are explained with reference to a data intake and query systemsimilar to the SPLUNK® ENTERPRISE system, the techniques are alsoapplicable to other types of data systems.

In the SPLUNK® ENTERPRISE system, machine-generated data is collectedand stored as “events,” where each event comprises a portion of themachine-generated data and is associated with a specific point in time.For example, events may be derived from “time series data,” where thetime series data comprises a sequence of data points (e.g., performancemeasurements from a computer system) that are associated with successivepoints in time. In general, each event can be associated with atimestamp that is derived from the raw data in the event, determinedthrough interpolation between temporally proximate events having knowntimestamps, determined based on other configurable rules for assigningtimestamps to events, etc.

Events can be derived from either “structured” or “unstructured” machinedata. In general, structured data has a predefined format, where dataitems with specific data formats are stored at predefined locations inthe data. For example, structured data may include data stored as fieldsin a database table. In contrast, unstructured data may not have apredefined format. This means that unstructured data can comprisevarious data items of different data types and that may be stored atdifferent locations within the data. For example, when the data sourceis an operating system log, an event can include one or more lines fromthe operating system log containing raw data that includes differenttypes of performance and diagnostic information associated with aspecific point in time.

Examples of components which may generate machine data from which eventscan be derived include, but are not limited to, web servers, applicationservers, databases, firewalls, routers, operating systems, and softwareapplications that execute on computer systems, mobile devices, sensors,etc. The data generated by such data sources can include, for exampleand without limitation, server log files, activity log files,configuration files, messages, network packet data, performancemeasurements, sensor measurements, etc.

The SPLUNK® ENTERPRISE system also facilitates using a flexible schemato specify how to extract information from the event data, where theflexible schema may be developed and redefined as needed. Note that aflexible schema may be applied to event data “on the fly,” when it isneeded (e.g., at search time, etc.), rather than at ingestion time ofthe data as in traditional database systems. Because the schema is notapplied to event data until it is needed (e.g., at search time, etc.),it may be referred to as a “late-binding schema.”

During operation, the SPLUNK® ENTERPRISE system starts with raw inputdata (e.g., one or more log files, a stream of network data, etc.). Thesystem divides this raw data into blocks, and parses the data to producetimestamped events. The system stores the timestamped events in a datastore, and enables users to run queries against the stored data toretrieve events that meet criteria specified in a query, such ascontaining certain keywords or having specific values in defined fields.In this context, the term “field” refers to a location in the event datacontaining a value for a specific data item.

As noted above, the SPLUNK® ENTERPRISE system facilitates applying alate-binding schema to event data while performing queries on events.One aspect of a late-binding schema is “extraction rules” that areapplied to data in the events to extract values for specific fields.More specifically, the extraction rules for a field can include one ormore instructions that specify how to extract a value for the field fromthe event data. An extraction rule can generally include any type ofinstruction for extracting values from data in events. In some cases, anextraction rule comprises a regular expression, in which case the ruleis referred to as a “regex rule.” In the SPLUNK® ENTERPRISE system, afield extractor may be configured to automatically generate extractionrules for certain fields in the events when the events are beingcreated, indexed, or stored, or possibly at a later time. Alternatively,a user may manually define extraction rules for fields using a varietyof techniques.

In contrast to a conventional schema for a database system, alate-binding schema is not defined at data ingestion time. Instead, thelate-binding schema can be developed on an ongoing basis until the timea query is actually executed. This means that extraction rules for thefields in a query may be provided in the query itself, or may be locatedduring execution of the query. Hence, as an analyst learns more aboutthe data in the events, the analyst can continue to refine thelate-binding schema by adding new fields, deleting fields, or modifyingthe field extraction rules for use the next time the schema is used bythe system. Because the SPLUNK® ENTERPRISE system maintains theunderlying raw data and uses late-binding schemas for searching the rawdata, it enables an analyst to investigate questions that arise as theanalyst learns more about the events.

In some embodiments, a common field name may be used to reference two ormore fields containing equivalent data items, even though the fields maybe associated with different types of events that possibly havedifferent data formats and different extraction rules. By enabling acommon field name to be used to identify equivalent fields fromdifferent types of events generated by different data sources, thesystem facilitates use of a “common information model” (CIM) across thedifferent data sources.

2.2. Data Server System

FIG. 1 depicts a block diagram of an example data intake and querysystem 100, similar to the SPLUNK® ENTERPRISE system. System 100includes one or more forwarders 104 that consume data from a variety ofinput data sources 102, and one or more indexers 106 that process andstore the data in one or more data stores 108. These forwarders andindexers can comprise separate computer systems, or may alternativelycomprise separate processes executing on one or more computer systems.

During operation, the forwarders 104 identify which indexers 106 receivedata collected from a data source 102 and forward the data to theappropriate indexers. Forwarders 104 can also perform operations on thedata before forwarding, including removing extraneous data, detectingtimestamps in the data, performing other data transformations (e.g.,parsing data, etc.), etc.

In general, distributing data across different indexers facilitatesparallel processing and fault tolerance. This parallel processing cantake place at data ingestion time because multiple indexers can processthe incoming data in parallel. The parallel processing can also takeplace at search time because multiple indexers can perform searches onthe data in parallel.

System 100 and the processes described below with reference to FIG. 1,FIG. 2, FIG. 3, FIG. 4, and FIG. 5 are further described in “ExploringSplunk Search Processing Language (SPL) Primer and Cookbook” by DavidCarasso, CITO Research, 2012, and in “Optimizing Data Analysis With aSemi-Structured Time Series Database” by Ledion Bitincka, ArchanaGanapathi, Stephen Sokin, and Steve Zhang, SLAML, 2010, each of which ishereby incorporated herein by reference in its entirety for allpurposes.

2.3. Source Types Overview

In general, a data intake and query system such as system 100 mayinclude a large number of configurable aspects that enable anadministrator or other user to customize the system for particular typesof data to be analyzed and to accommodate other user preferences.Configuration data corresponding to some or all of these aspects may bestored in one or more configuration files or other storage mechanismsthat are accessible to various components of the system 100. Suchconfiguration data may collectively define how a system 100 collects andprocesses certain types of data, indexes data, performs searches onstored data, performs data backups, and performs many other features ofthe system 100.

In an embodiment, one configurable aspect of a system 100 is referred toherein as “source types.” In this context, a source type refers to aparticular type and format of input data that can be consumed by asystem 100 for analysis. As one example, one source type may correspondto a particular type of log data generated by a web server and in whichdiagnostic and other information related to the operation of the webserver is formatted and stored in a particular manner. Other sourcetypes may correspond to other types of data, each of which may generatedby different devices or applications and formatted according to anynumber of different formats.

In general, each source type of a system 100 may or may not be unique toa particular type of device or application. For example, two or moredifferent types of applications or devices may produce log files thatconform to the same format and thus may correspond to the same sourcetype. Each of the applications may, for example, incorporate the samelogging tool that produces a similarly formatted log file in differentcontexts. Similarly, a single application or device may generate two ormore different types of data corresponding to two or more differentsource types. For example, a single application may generate separatelog files conforming to different formats, or may generate both logfiles and network data.

In an embodiment, configuration data related to source types of a system100 includes one or more source type “definitions.” In general, eachsource type definition relates to one or more source types and includesone or more properties that define how various components of a system100 handle input data and events associated with that source type. Forexample, one source type definition may correspond to log data generatedby a particular type of web server and include one or more propertiesthat define how system 100 interacts with the log data. Another sourcetype definition, for example, may include different properties defininghow system 100 processes output generated by one or more types ofnetwork devices.

In an embodiment, the properties of a source type definition generallycomprise a collection of attribute-value pairs. Each attribute mayrelate to a different aspect of how system 100 interacts with data of aparticular source type. A value associated with a particular attributemay specify one or more definitions, rules, or other values related tothe corresponding attribute. The attribute-value pairs comprising asource type definition may indicate, among other information, how asystem 100 segments input data associated with a particular source typeinto events, locates or otherwise defines timestamps to be associatedwith each generated event, and extraction rules for defining fieldswithin the resulting events.

In an embodiment, a data intake and query system 100 may include anumber of pre-defined source type definitions corresponding to varioustypes of input data that commonly may be analyzed. For example, aprovider of the data intake and query system 100 may pre-define one ormore source type definitions corresponding to common types of operatingsystem logs, router logs, and other data generated by widely usedapplications and devices. According to various embodiments describedherein, a data intake and query system 100 may further enable users tocreate, modify, and delete source type definitions to customize a system100 for virtually any type of input data a user may desire to analyze.

In one embodiment, each source type definition may be associated with alabel or other identifier that is used to identify the definitionthroughout a system 100. For example, this label may be stored in ametadata field or otherwise associated with each event created from datareceived from an input source. Based on the label accompanying eachevent, various components of the system 100 can locate a correspondingsource type definition in configuration data accessible to thecomponents when interacting with particular events.

In one embodiment, each source type definition may be stored in one ormore configuration files. For example, a label and one or moreattribute-value pairs associated with a source type definition may bestored in one or more text files, database tables, or other storagemechanism. Each source type definition may comprise data stored acrossone or more separate configuration files, and each configuration filemay include data related to one or more separate source typedefinitions. One or more copies of these configuration files may existacross various components of a data intake and query system 100.Referring to FIG. 1, for example, one or more forwarders 104, indexers106, and search heads 110 may each store local copies of one or more ofthese configuration files. In other embodiments, one or more componentsof a system 100 may refer to a shared copy of one or more configurationfiles.

The following lines represent a portion of an example configuration fileincluding one or more source type definitions:

[default]

CHARSET=AUTO

DATETIME_CONFIG=\etc\datetime.xml

MAX_DAYS_HENCE=2

MAX_DAYS_AGO=2000

MAX_DIFF_SECS_AGO=3600

MAX_DIFF_SECS_HENCE=604800

MAX_TIMESTAMP_LOOKAHEAD=128

MAX_EVENTS=256

LEARN_SOURCETYPE=true

In the example above, the first line provides a label for the sourcetype definition. In this example, the label provided is “default,”indicating that this source type definition represents defaultproperties which may, for example, apply across all input data unlessoverridden by one or more source type-specific definitions.

The lines following the source type definition label in the exampleabove illustrate several example attribute-value pairs. In the example,each attribute-value pair includes an attribute name and a valueseparate by an “=” character. Some of the attribute-value pairs mayrelate to how components of a data intake and query system parse inputdata. For example, a value of “AUTO” specified for the CHARSET attributemay indicate that the system is to auto-detect a type of encodingassociated with input data. As another example, the value specified forthe MAX_EVENTS attribute may specify a maximum number of input lines toadd to any particular event.

Other attribute-value pairs may relate to how timestamps are generatedfor each event. The value associated with the DATETIME_CONFIG attributemay, for example, provide a reference to a file that configures atimestamp extractor. As another example, the values associated with theMAX_DAYS_HENCE, MAX_DAYS_AGO, MAX_DIFF_SECS_AGO, MAX_DIFF_SECS_HENCE,and MAX_TIMESTAMP_LOOKAHEAD attributes may determine how dateinformation included in input data is interpreted to generate timestampsfor events.

The following lines represent another example portion of a configurationfile illustrating a source type-specific definition:

[log 4j]

BREAK_ONLY_BEFORE=\d\d?:\d\d:\d\d

category=Application

description=Output produced by any Java 2 Enterprise Edition (J2EE)application server using log 4j

As indicated by the value for the description attribute, the examplesource type configuration above may relate to output produced by a Java2 Enterprise Edition (J2EE) application server using log 4j, a popularframework for generating log statements. The first line provides a labelof “log 4j” for the source type definition. The second line includes anattribute name “BREAK_ONLY_BEFORE” and a regular expression value whichmay, for example, indicate that the system 100 is to create a separateevent when it encounters a new line in input data that matches theregular expression. The category attribute may, for example, provide avalue used to describe the category of source type for use in variousinterfaces of the system. The example source type definition excerptsillustrated above are provided for illustrative purposes only. Ingeneral, source type definitions may be formatted in any manner.

2.4. Data Ingestion

FIG. 2 depicts a flow chart illustrating an example data flow within adata intake and query system 100, in accordance with the disclosedembodiments. The data flow illustrated in FIG. 2 is provided forillustrative purposes only; one or more of the steps of the processesillustrated in FIG. 2 may be removed or the ordering of the steps may bechanged. Furthermore, for the purposes of illustrating a clear example,one or more particular system components is described as performingvarious operations during each of the data flow stages. For example, aforwarder is described as receiving and processing data during an inputphase, an indexer is described as parsing and indexing data duringparsing and indexing phases, and a search head is described asperforming a search query during a search phase. However, it is notedthat other system arrangements and distributions of the processing stepsacross system components may be used.

2.4.1. Input

At block 202, a forwarder receives data from an input source. Aforwarder, for example, initially may receive the data as a raw datastream generated by the input source. For example, a forwarder mayreceive a data stream from a log file generated by an applicationserver, from a stream of network data from a network device, or from anyother source of data. In one embodiment, a forwarder receives the rawdata and may segment the data stream into “blocks,” possibly of auniform data size, to facilitate subsequent processing steps.

At block 204, a forwarder or other system component annotates each blockgenerated from the raw data with one or more metadata fields. Thesemetadata fields may, for example, provide information related to thedata block as a whole and which apply to each event that is subsequentlyderived from the data block, as described in more detail below. Forexample, the metadata fields may include separate fields specifying eachof a host, a source, and a source type related to the data block. A hostfield, for example, may contain a value identifying a host name or IPaddress of a device that generated the data. A source field may containa value identifying a source of the data, such as a pathname of a fileor a protocol and port related to received network data. A source typefield may contain a value specifying a particular source type label forthe data. Additional metadata fields may also be included during theinput phase, such as a character encoding of the data if known, andpossibly other values that provide information relevant to laterprocessing steps. In an embodiment, a forwarder forwards the data toanother system component for further processing, typically forwardingthe annotated data blocks to an indexer.

2.4.2. Parsing

At block 206, an indexer receives data blocks from a forwarder andparses the data to organize the data into events. In an embodiment, toorganize the data into events, an indexer may determine a source typeassociated with each data block (e.g., by extracting a source type labelfrom the metadata fields associated with the data block) and refer to asource type configuration corresponding to the identified source type.The source type definition may include one or more properties thatindicate to the indexer what are the boundaries of events in the data.In general, these properties may include regular expression-based rulesor delimiter rules where, for example, event boundaries may be indicatedby predefined characters or character strings. These predefinedcharacters may include punctuation marks or other special charactersincluding, for example, carriage returns, tabs, spaces, or line breaks.If a source type for the data is unknown to the indexer, an indexer mayinfer a source type for the data by examining the structure of the dataand apply an inferred source type definition to the data to create theevents.

At block 208, the indexer determines a timestamp for each event. Similarto the process for creating events, an indexer may again refer to asource type definition associated with the data to locate one or moreproperties that indicate instructions for determining a timestamp foreach event. The properties may, for example, instruct an indexer toextract a time value from a portion of data in the event, to interpolatetime values based on timestamps associated with temporally proximateevents, to create a timestamp based on a time the event data wasreceived or generated, or based on any other rules for determiningtimestamps.

At block 210, the indexer associates with each event one or moremetadata fields including a field containing the timestamp determinedfor the event. These metadata fields may include a number of “defaultfields” that are associated with all events, and may also include onemore custom fields as defined by a user. Similar to the metadata fieldsassociated with the data blocks at block 204, the default metadatafields associated with each event may include a host, source, and sourcetype field in addition to a field storing the timestamp.

At block 212, an indexer may optionally apply one or moretransformations to data included in the events created at block 206. Forexample, such transformations can include removing a portion of an event(e.g., a portion used to define event boundaries, other extraneous text,etc.), masking a portion of an event (e.g., masking a credit cardnumber), or removing redundant portions of an event. The transformationsapplied to event data may, for example, be specified in one or moreconfiguration files and referenced by one or more source typedefinitions.

2.4.3. Indexing

At blocks 214 and 216, an indexer can optionally generate a keywordindex to facilitate fast keyword searching for event data. To build akeyword index, at block 214, the indexer identifies a set of keywords ineach event. At block 216, the indexer includes the identified keywordsin an index, which associates each stored keyword with referencepointers to events containing that keyword (or to locations withinevents where that keyword is located, other location identifiers, etc.).When an indexer subsequently receives a keyword-based query, the indexercan access the keyword index to quickly identify events containing thekeyword.

In some embodiments, the keyword index may include entries forname-value pairs found in events, where a name-value pair can include apair of keywords connected by a symbol, such as an equals sign or colon.In this way, events containing these name-value pairs can be quicklylocated. In some embodiments, fields can automatically be generated forsome or all of the name-value pairs at the time of indexing. Forexample, if the string “dest=10.0.1.2” is found in an event, a fieldnamed “dest” may be created for the event, and assigned a value of“10.0.1.2”.

At block 218, the indexer stores the events in a data store, where atimestamp can be stored with each event to facilitate searching forevents based on a time range. In one embodiment, the stored events areorganized into “buckets,” where each bucket stores events associatedwith a specific time range based on the timestamps associated with eachevent. This may not only improve time-based searching, but also allowfor events with recent timestamps, which may have a higher likelihood ofbeing accessed, to be stored in faster memory to facilitate fasterretrieval. For example, buckets containing the most recent events can bestored as flash memory instead of on hard disk.

Each indexer 106 may be responsible for storing and searching a subsetof the events contained in a corresponding data store 108. Bydistributing events among the indexers and data stores, the indexers cananalyze events for a query in parallel, for example, using map-reducetechniques, wherein each indexer returns partial responses for a subsetof events to a search head that combines the results to produce ananswer for the query. By storing events in buckets for specific timeranges, an indexer may further optimize searching by looking only inbuckets for time ranges that are relevant to a query.

Moreover, events and buckets can also be replicated across differentindexers and data stores to facilitate high availability and disasterrecovery as is described in U.S. patent application Ser. No. 14/266,812,filed on 30 Apr. 2014, and in U.S. patent application Ser. No.14/266,817, also filed on 30 Apr. 2014, each of which is herebyincorporated by reference in its entirety for all purposes.

2.5. Query Processing

FIG. 3 is a flow diagram that illustrates an example process that asearch head and one or more indexers may perform during a search query.At block 302, a search head receives a search query from a client. Atblock 304, the search head analyzes the search query to determine whatportions can be delegated to indexers and what portions can be executedlocally by the search head. At block 306, the search head distributesthe determined portions of the query to the appropriate indexers.

At block 308, the indexers to which the query was distributed searchtheir data stores for events that are responsive to the query. Todetermine which events are responsive to the query, the indexer searchesfor events that match the criteria specified in the query. This criteriacan include matching keywords or specific values for certain fields. Insearches that use a late-binding schema, the searching operations atblock 308 may involve using the late-binding schema to extract valuesfor specified fields from events at the time the query is processed. Inan embodiment, one or more rules for extracting field values may bespecified as part of a source type definition. The indexers may theneither send the relevant events back to the search head, or use theevents to calculate a partial result, and send the partial result backto the search head.

At block 310, the search head combines the partial results and/or eventsreceived from the indexers to produce a result for the query. Thisresult may comprise different types of data depending on what the queryrequested. For example, the results can include a listing of matchingevents returned by the query, or some type of visualization of the datafrom the returned events. In another example, the final result caninclude one or more calculated values derived from the matching events.

The results generated by the system 100 can be returned to a clientusing different techniques. For example, one technique streams resultsback to a client in real-time as they are identified. Another techniquewaits to report the results to the client until a complete set ofresults is ready to return to the client. Yet another technique streamsinterim results back to the client in real-time until a complete set ofresults is ready, and then returns the complete set of results to theclient. In another technique, certain results are stored as “searchjobs” and the client may retrieve the results by referring the searchjobs.

The search head can also perform various operations to make the searchmore efficient. For example, before the search head begins execution ofa query, the search head can determine a time range for the query and aset of common keywords that all matching events include. The search headmay then use these parameters to query the indexers to obtain a supersetof the eventual results. Then, during a filtering stage, the search headcan perform field-extraction operations on the superset to produce areduced set of search results.

2.6. Field Extraction

FIG. 4 illustrates an example of applying extraction rules to a searchquery received from a client. At the start of the process, a searchquery 402 is received at a query processor 404. Query processor 404includes various mechanisms for processing a query and may reside in asearch head 110 and/or an indexer 106. Note that the example searchquery 402 illustrated in FIG. 4 is expressed in Search ProcessingLanguage (SPL), which is used in conjunction with the SPLUNK® ENTERPRISEsystem. SPL is a pipelined search language in which a set of inputs isoperated on by a first command in a command line, and then a subsequentcommand following the pipe symbol “|” operates on the results producedby the first command, and so on for additional commands. Search query402 can also be expressed in other query languages, such as theStructured Query Language (“SQL”) or any other query language.

In response to receiving search query 402, query processor 404determines that search query 402 refers to two fields: “IP” and“target.” Query processor 404 also determines that the values for the“IP” and “target” fields have not already been extracted from eventsstored in a data store 414, and consequently determines that queryprocessor 404 can use extraction rules to extract values for the fields.Hence, query processor 404 performs a lookup for the extraction rules ina rule base 406. For example, rule base 406 may include a source typedefinition, where the source type definition includes extraction rulesfor various different source types. The query processor 404 obtainsextraction rules 408-409, wherein extraction rule 408 specifies how toextract a value for the “IP” field from an event, and extraction rule409 specifies how to extract a value for the “target” field from anevent. As is illustrated in FIG. 4, extraction rules 408-409 cancomprise regular expressions that specify how to extract values for therelevant fields. Such regular expression-based extraction rules are alsoreferred to as “regex rules.” In addition to specifying how to extractfield values, the extraction rules may also include instructions forderiving a field value by performing a function on a character string orvalue retrieved by the extraction rule. For example, a transformationrule may truncate a character string, or convert the character stringinto a different data format. In some cases, the query itself canspecify one or more extraction rules.

Next, query processor 404 sends extraction rules 408-409 to a fieldextractor 412, which applies extraction rules 408-409 to events 416-418in a data store 414. Note that data store 414 can include one or moredata stores, and extraction rules 408-409 can be applied to largenumbers of events in data store 414, and are not meant to be limited tothe three events 416-417 illustrated in FIG. 4. Moreover, the queryprocessor 414 can instruct field extractor 412 to apply the extractionrules to all the events in a data store 414, or to a subset of theevents that have been filtered based on some criteria.

Next, field extractor 412 applies extraction rule 408 for the firstcommand “Search IP=“10*” to events in data store 414 including events416-418. Extraction rule 408 is used to extract values for the IPaddress field from events in data store 414 by looking for a pattern ofone or more digits, followed by a period, followed again by one or moredigits, followed by another period, followed again by one or moredigitals, followed by another period, and followed again by one or moredigits. Next, field extractor 412 returns field values 420 to queryprocessor 404, which uses the criterion IP=“10*” to look for IPaddresses that start with “10”. Note that events 416 and 417 match thiscriterion, but event 418 does not, so the result set for the firstcommand includes events 416-417.

Query processor 404 then sends events 416-417 to the next command “statscount target.” To process this command, query processor 404 causes fieldextractor 412 to apply extraction rule 409 to events 416-417. Extractionrule 409 is used to extract values for the target field for events416-417 by skipping the first four commas in events 416-417, and thenextracting all of the following characters until a comma or period isreached. Next, field extractor 412 returns field values 421 to queryprocessor 404, which executes the command “stats count target” to countthe number of unique values contained in the target fields, which inthis example produces the value “2” that is returned as a final result422 for the query.

Note that query results can be returned to a client, a search head, orany other system component for further processing. In general, queryresults may include a set of one or more events, a set of one or morevalues obtained from the events, a subset of the values, statisticscalculated based on the values, a report containing the values, or avisualization, such as a graph or chart, generated from the values.

2.7. Example Search Screen

FIG. 6A illustrates an example search screen 600 in accordance with thedisclosed embodiments. Search screen 600 includes a search bar 602 thataccepts user input in the form of a search string. It also includes atime range picker 612 that enables the user to specify a time range forthe search. For “historical searches” the user can select a specifictime range, or alternatively a relative time range, such as “today,”“yesterday” or “last week.” For “real-time searches,” the user canselect the size of a preceding time window to search for real-timeevents. Search screen 600 also initially displays a “data summary”dialog as is illustrated in FIG. 6B that enables the user to selectdifferent sources for the event data, for example by selecting specifichosts and log files.

After the search is executed, the search screen 600 can display theresults through search results tabs 604, wherein search results tabs 604includes: an “events tab” that displays various information about eventsreturned by the search; a “statistics tab” that displays statisticsabout the search results; and a “visualization tab” that displaysvarious visualizations of the search results. The events tab illustratedin FIG. 6A displays a timeline graph 605 that graphically illustratesthe number of events that occurred in one-hour intervals over theselected time range. It also displays an events list 608 that enables auser to view the raw data in each of the returned events. Itadditionally displays a fields sidebar 606 that includes statisticsabout occurrences of specific fields in the returned events, including“selected fields” that are pre-selected by the user, and “interestingfields” that are automatically selected by the system based onpre-specified criteria.

2.8. Acceleration Technique

The above-described system provides significant flexibility by enablinga user to analyze massive quantities of minimally processed performancedata “on the fly” at search time instead of storing pre-specifiedportions of the performance data in a database at ingestion time. Thisflexibility enables a user to see correlations in the performance dataand perform subsequent queries to examine interesting aspects of theperformance data that may not have been apparent at ingestion time.

However, performing extraction and analysis operations at search timecan involve a large amount of data and require a large number ofcomputational operations, which can cause considerable delays whileprocessing the queries. Fortunately, a number of acceleration techniqueshave been developed to speed up analysis operations performed at searchtime. These techniques include: (1) performing search operations inparallel by formulating a search as a map-reduce computation; (2) usinga keyword index; (3) using a high performance analytics store; and (4)accelerating the process of generating reports. These techniques aredescribed in more detail below.

2.8.1. Map-Reduce Technique

To facilitate faster query processing, a query can be structured as amap-reduce computation, wherein the “map” operations are delegated tothe indexers, while the corresponding “reduce” operations are performedlocally at the search head. For example, FIG. 5 illustrates how a searchquery 502 received from a client at a search head 110 can split into twophases, including: (1) a “map phase” comprising subtasks 504 (e.g., dataretrieval or simple filtering) that may be performed in parallel and are“mapped” to indexers 106 for execution, and (2) a “reduce phase”comprising a merging operation 506 to be executed by the search headwhen the results are ultimately collected from the indexers.

During operation, upon receiving search query 502, a search head 110modifies search query 502 by substituting “stats” with “prestats” toproduce search query 504, and then distributes search query 504 to oneor more distributed indexers, which are also referred to as “searchpeers.” Note that search queries may generally specify search criteriaor operations to be performed on events that meet the search criteria.Search queries may also specify field names, as well as search criteriafor the values in the fields or operations to be performed on the valuesin the fields. Moreover, the search head may distribute the full searchquery to the search peers as is illustrated in FIG. 3, or mayalternatively distribute a modified version (e.g., a more restrictedversion) of the search query to the search peers. In this example, theindexers are responsible for producing the results and sending them tothe search head. After the indexers return the results to the searchhead, the search head performs the merging operations 506 on theresults. Note that by executing the computation in this way, the systemeffectively distributes the computational operations while minimizingdata transfers.

2.8.2. Keyword Index

As described above with reference to the flow charts in FIG. 2 and FIG.3, data intake and query system 100 can construct and maintain one ormore keyword indices to facilitate rapidly identifying events containingspecific keywords. This can greatly speed up the processing of queriesinvolving specific keywords. As mentioned above, to build a keywordindex, an indexer first identifies a set of keywords. Then, the indexerincludes the identified keywords in an index, which associates eachstored keyword with references to events containing that keyword, or tolocations within events where that keyword is located. When an indexersubsequently receives a keyword-based query, the indexer can access thekeyword index to quickly identify events containing the keyword.

2.8.3. High Performance Analytics Store

To speed up certain types of queries, some embodiments of system 100make use of a high performance analytics store, which is referred to asa “summarization table,” that contains entries for specific field-valuepairs. Each of these entries keeps track of instances of a specificvalue in a specific field in the event data and includes references toevents containing the specific value in the specific field. For example,an example entry in a summarization table can keep track of occurrencesof the value “94107” in a “ZIP code” field of a set of events, whereinthe entry includes references to all of the events that contain thevalue “94107” in the ZIP code field. This enables the system to quicklyprocess queries that seek to determine how many events have a particularvalue for a particular field, because the system can examine the entryin the summarization table to count instances of the specific value inthe field without having to go through the individual events or doextractions at search time. Also, if the system needs to process allevents that have a specific field-value combination, the system can usethe references in the summarization table entry to directly access theevents to extract further information without having to search all ofthe events to find the specific field-value combination at search time.

In some embodiments, the system maintains a separate summarization tablefor each of the above-described time-specific buckets that stores eventsfor a specific time range, wherein a bucket-specific summarization tableincludes entries for specific field-value combinations that occur inevents in the specific bucket. Alternatively, the system can maintain aseparate summarization table for each indexer, wherein theindexer-specific summarization table only includes entries for theevents in a data store that is managed by the specific indexer.

The summarization table can be populated by running a “collection query”that scans a set of events to find instances of a specific field-valuecombination, or alternatively instances of all field-value combinationsfor a specific field. A collection query can be initiated by a user, orcan be scheduled to occur automatically at specific time intervals. Acollection query can also be automatically launched in response to aquery that asks for a specific field-value combination.

In some cases, the summarization tables may not cover all of the eventsthat are relevant to a query. In this case, the system can use thesummarization tables to obtain partial results for the events that arecovered by summarization tables, but may also have to search throughother events that are not covered by the summarization tables to produceadditional results. These additional results can then be combined withthe partial results to produce a final set of results for the query.This summarization table and associated techniques are described in moredetail in U.S. Pat. No. 8,682,925, issued on Mar. 25, 2014.

2.8.4. Accelerating Report Generation

In some embodiments, a data server system such as the SPLUNK® ENTERPRISEsystem can accelerate the process of periodically generating updatedreports based on query results. To accelerate this process, asummarization engine automatically examines the query to determinewhether generation of updated reports can be accelerated by creatingintermediate summaries. (This is possible if results from preceding timeperiods can be computed separately and combined to generate an updatedreport. In some cases, it is not possible to combine such incrementalresults, for example where a value in the report depends onrelationships between events from different time periods.) If reportscan be accelerated, the summarization engine periodically generates asummary covering data obtained during a latest non-overlapping timeperiod. For example, where the query seeks events meeting a specifiedcriteria, a summary for the time period includes only events within thetime period that meet the specified criteria. Similarly, if the queryseeks statistics calculated from the events, such as the number ofevents that match the specified criteria, then the summary for the timeperiod includes the number of events in the period that match thespecified criteria.

In parallel with the creation of the summaries, the summarization engineschedules the periodic updating of the report associated with the query.During each scheduled report update, the query engine determines whetherintermediate summaries have been generated covering portions of the timeperiod covered by the report update. If so, then the report is generatedbased on the information contained in the summaries. Also, if additionalevent data has been received and has not yet been summarized, and isrequired to generate the complete report, the query can be run on thisadditional event data. Then, the results returned by this query on theadditional event data, along with the partial results obtained from theintermediate summaries, can be combined to generate the updated report.This process is repeated each time the report is updated. Alternatively,if the system stores events in buckets covering specific time ranges,then the summaries can be generated on a bucket-by-bucket basis. Notethat producing intermediate summaries can save the work involved inre-running the query for previous time periods, so only the newer eventdata needs to be processed while generating an updated report. Thesereport acceleration techniques are described in more detail in U.S. Pat.No. 8,589,403, issued on 19 Nov. 2013, and U.S. Pat. No. 8,412,696,issued on 2 Apr. 2011.

2.9. Security Features

The SPLUNK® ENTERPRISE platform provides various schemas, dashboards andvisualizations that make it easy for developers to create applicationsto provide additional capabilities. One such application is the SPLUNK®APP FOR ENTERPRISE SECURITY, which performs monitoring and alertingoperations and includes analytics to facilitate identifying both knownand unknown security threats based on large volumes of data stored bythe SPLUNK® ENTERPRISE system. This differs significantly fromconventional Security Information and Event Management (SIEM) systemsthat lack the infrastructure to effectively store and analyze largevolumes of security-related event data. Traditional SIEM systemstypically use fixed schemas to extract data from pre-definedsecurity-related fields at data ingestion time, wherein the extracteddata is typically stored in a relational database. This data extractionprocess (and associated reduction in data size) that occurs at dataingestion time inevitably hampers future incident investigations, whenall of the original data may be needed to determine the root cause of asecurity issue, or to detect the tiny fingerprints of an impendingsecurity threat.

In contrast, the SPLUNK® APP FOR ENTERPRISE SECURITY system stores largevolumes of minimally processed security-related data at ingestion timefor later retrieval and analysis at search time when a live securitythreat is being investigated. To facilitate this data retrieval process,the SPLUNK® APP FOR ENTERPRISE SECURITY provides pre-specified schemasfor extracting relevant values from the different types ofsecurity-related event data, and also enables a user to define suchschemas.

The SPLUNK® APP FOR ENTERPRISE SECURITY can process many types ofsecurity-related information. In general, this security-relatedinformation can include any information that can be used to identifysecurity threats. For example, the security-related information caninclude network-related information, such as IP addresses, domain names,asset identifiers, network traffic volume, uniform resource locatorstrings, and source addresses. (The process of detecting securitythreats for network-related information is further described in U.S.patent application Ser. Nos. 13/956,252, and 13/956,262.)Security-related information can also include endpoint information, suchas malware infection data and system configuration information, as wellas access control information, such as login/logout information andaccess failure notifications. The security-related information canoriginate from various sources within a data center, such as hosts,virtual machines, storage devices and sensors. The security-relatedinformation can also originate from various sources in a network, suchas routers, switches, email servers, proxy servers, gateways, firewallsand intrusion-detection systems.

During operation, the SPLUNK® APP FOR ENTERPRISE SECURITY facilitatesdetecting so-called “notable events” that are likely to indicate asecurity threat. These notable events can be detected in a number ofways: (1) an analyst can notice a correlation in the data and canmanually identify a corresponding group of one or more events as“notable;” or (2) an analyst can define a “correlation search”specifying criteria for a notable event, and every time one or moreevents satisfy the criteria, the application can indicate that the oneor more events are notable. An analyst can alternatively select apre-defined correlation search provided by the application. Note thatcorrelation searches can be run continuously or at regular intervals(e.g., every hour) to search for notable events. Upon detection, notableevents can be stored in a dedicated “notable events index,” which can besubsequently accessed to generate various visualizations containingsecurity-related information. Also, alerts can be generated to notifysystem operators when important notable events are discovered.

The SPLUNK® APP FOR ENTERPRISE SECURITY provides various visualizationsto aid in discovering security threats, such as a “key indicators view”that enables a user to view security metrics of interest, such as countsof different types of notable events. For example, FIG. 7A illustratesan example key indicators view 700 that comprises a dashboard, which candisplay a value 701, for various security-related metrics, such asmalware infections 702. It can also display a change in a metric value703, which indicates that the number of malware infections increased by63 during the preceding interval. Key indicators view 700 additionallydisplays a histogram panel 704 that displays a histogram of notableevents organized by urgency values, and a histogram of notable eventsorganized by time intervals. This key indicators view is described infurther detail in pending U.S. patent application Ser. No. 13/956,338filed Jul. 31, 2013.

These visualizations can also include an “incident review dashboard”that enables a user to view and act on “notable events.” These notableevents can include: (1) a single event of high importance, such as anyactivity from a known web attacker; or (2) multiple events thatcollectively warrant review, such as a large number of authenticationfailures on a host followed by a successful authentication. For example,FIG. 7B illustrates an example incident review dashboard 710 thatincludes a set of incident attribute fields 711 that, for example,enables a user to specify a time range field 712 for the displayedevents. It also includes a timeline 713 that graphically illustrates thenumber of incidents that occurred in one-hour time intervals over theselected time range. It additionally displays an events list 714 thatenables a user to view a list of all of the notable events that matchthe criteria in the incident attributes fields 711. To facilitateidentifying patterns among the notable events, each notable event can beassociated with an urgency value (e.g., low, medium, high, critical),which is indicated in the incident review dashboard. The urgency valuefor a detected event can be determined based on the severity of theevent and the priority of the system component associated with theevent. The incident review dashboard is described further in“http://docs.splunk.com/Documentation/PCI/2.1.1/User/IncidentReviewdashboard.”

2.10. Data Center Monitoring

As mentioned above, the SPLUNK® ENTERPRISE platform provides variousfeatures that make it easy for developers to create variousapplications. One such application is the SPLUNK® APP FOR VMWARE®, whichperforms monitoring operations and includes analytics to facilitatediagnosing the root cause of performance problems in a data center basedon large volumes of data stored by the SPLUNK® ENTERPRISE system.

This differs from conventional data-center-monitoring systems that lackthe infrastructure to effectively store and analyze large volumes ofperformance information and log data obtained from the data center. Inconventional data-center-monitoring systems, this performance data istypically pre-processed prior to being stored, for example by extractingpre-specified data items from the performance data and storing them in adatabase to facilitate subsequent retrieval and analysis at search time.However, the rest of the performance data is not saved and isessentially discarded during pre-processing. In contrast, the SPLUNK®APP FOR VMWARE® stores large volumes of minimally processed performanceinformation and log data at ingestion time for later retrieval andanalysis at search time when a live performance issue is beinginvestigated.

The SPLUNK® APP FOR VMWARE® can process many types ofperformance-related information. In general, this performance-relatedinformation can include any type of performance-related data and logdata produced by virtual machines and host computer systems in a datacenter. In addition to data obtained from various log files, thisperformance-related information can include values for performancemetrics obtained through an application programming interface (API)provided as part of the vSphere Hypervisor™ system distributed byVMware, Inc. of Palo Alto, Calif. For example, these performance metricscan include: (1) CPU-related performance metrics; (2) disk-relatedperformance metrics; (3) memory-related performance metrics; (4)network-related performance metrics; (5) energy-usage statistics; (6)data-traffic-related performance metrics; (7) overall systemavailability performance metrics; (8) cluster-related performancemetrics; and (9) virtual machine performance statistics. For moredetails about such performance metrics, please see U.S. patent Ser. No.14/167,316 filed 29 Jan. 2014, which is hereby incorporated herein byreference. Also, see “vSphere Monitoring and Performance,” Update 1,vSphere 5.5, EN-001357-00,http://pubs.vmware.com/vsphere-55/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-551-monitoring-performance-guide.pdf.

To facilitate retrieving information of interest from performance dataand log files, the SPLUNK® APP FOR VMWARE® provides pre-specifiedschemas for extracting relevant values from different types ofperformance-related event data, and also enables a user to define suchschemas.

The SPLUNK® APP FOR VMWARE® additionally provides various visualizationsto facilitate detecting and diagnosing the root cause of performanceproblems. For example, one such visualization is a “proactive monitoringtree” that enables a user to easily view and understand relationshipsamong various factors that affect the performance of a hierarchicallystructured computing system. This proactive monitoring tree enables auser to easily navigate the hierarchy by selectively expanding nodesrepresenting various entities (e.g., virtual centers or computingclusters) to view performance information for lower-level nodesassociated with lower-level entities (e.g., virtual machines or hostsystems). Example node-expansion operations are illustrated in FIG. 7C,wherein nodes 733 and 734 are selectively expanded. Note that nodes731-739 can be displayed using different patterns or colors to representdifferent performance states, such as a critical state, a warning state,a normal state or an unknown/offline state. The ease of navigationprovided by selective expansion in combination with the associatedperformance-state information enables a user to quickly diagnose theroot cause of a performance problem. The proactive monitoring tree isdescribed in further detail in U.S. patent application Ser. No.14/235,490 filed on 15 Apr. 2014, which is hereby incorporated herein byreference for all possible purposes.

The SPLUNK® APP FOR VMWARE® also provides a user interface that enablesa user to select a specific time range and then view heterogeneous data,comprising events, log data and associated performance metrics, for theselected time range. For example, the screen illustrated in FIG. 7Ddisplays a listing of recent “tasks and events” and a listing of recent“log entries” for a selected time range above a performance-metric graphfor “average CPU core utilization” for the selected time range. Notethat a user is able to operate pull-down menus 742 to selectivelydisplay different performance metric graphs for the selected time range.This enables the user to correlate trends in the performance-metricgraph with corresponding event and log data to quickly determine theroot cause of a performance problem. This user interface is described inmore detail in U.S. patent application Ser. No. 14/167,316 filed on 29Jan. 2014, which is hereby incorporated herein by reference for allpossible purposes.

2.11. Cloud-Based System Overview

The example data intake and query system 100 described in reference toFIG. 1 comprises several system components, including one or moreforwarders, indexers, and search heads. In some environments, a user ofa data intake and query system 100 may install and configure, oncomputing devices owned and operated by the user, one or more softwareapplications that implement some or all of these system components. Forexample, a user may install a software application on server computersowned by the user and configure each server to operate as one or more ofa forwarder, an indexer, a search head, etc. This arrangement generallymay be referred to as an “on-premises” solution, meaning the system 100is installed and operates on computing devices directly controlled bythe user of the system. Some users may prefer an on-premises solutionsince it may provide a greater level of control over the configurationof certain aspects of the system. However, other users may insteadprefer an arrangement in which the user is not directly responsible forproviding and managing the computing devices upon which variouscomponents of system 100 operate.

In one embodiment, to provide an alternative to an entirely on-premisesenvironment for system 100, one or more of the components of a dataintake and query system instead may be provided as a cloud-basedservice. In this context, a cloud-based service refers to a servicehosted by one more computing resources that are accessible to end usersover a network, for example, by using a web browser or other applicationon a client device to interface with the remote computing resources. Forexample, a service provider may provide a cloud-based data intake andquery system by managing computing resources configured to implementvarious aspects of the system (e.g., forwarders, indexers, search heads,etc.) and providing access to the system to end users via a network.Typically, a user may pay a subscription or other fee to use such aservice, and each subscribing user to the cloud-based service may beprovided with an account that enables the user to configure a customizedcloud-based system based on the user's preferences.

FIG. 8 illustrates a block diagram of an example cloud-based data intakeand query system. Similar to the system of FIG. 1, the networkedcomputer system 800 includes input data sources 102 and forwarders 104.In the example system 800 of FIG. 8, one or more forwarders 104 andclient devices 802 are coupled to a cloud-based data intake and querysystem 806 via one or more networks 804. Network 804 broadly representsone or more LANs, WANs, cellular networks, and/or internetworks usingany of wired, wireless, terrestrial microwave, satellite links, etc.,and may include the public Internet, and is used by client devices 802and forwarders 104 to access the system 800. Similar to the system of100, each of the forwarders 104 may be configured to receive data froman input source and to forward the data to other components of thesystem 806 for further processing.

In an embodiment, a cloud-based data intake and query system 806 maycomprise a plurality of system instances 808. In general, each systeminstance 808 may include one or more computing resources managed by aprovider of the cloud-based system 800 made available to a particularsubscriber. The computing resources comprising a system instance 808may, for example, include one or more servers or other devicesconfigured to implement one or more forwarders, indexers, search heads,and other components of a data intake and query system, similar tosystem 100. As indicated above, a subscriber may use a web browser orother application of a client device 802 to access a web portal or otherinterface that enables the subscriber to configure an instance 808.

Providing a data intake and query system as described in reference tosystem 100 as a cloud-based service presents a number of challenges. Asdescribed in reference to the data flow illustrated in FIG. 2, forexample, each of the components of a system 100 (e.g., forwarders,indexers and search heads) may at times refer to various configurationfiles stored locally at each component. These configuration filestypically may involve some level of user configuration to accommodateparticular types of data a user desires to analyze and to account forother user preferences. However, in a cloud-based service context, userstypically may not have direct access to the underlying computingresources implementing the various system components (e.g., thecomputing resources comprising each system instance 808). Thus maydesire to make such configurations indirectly, for example, using one ormore web-based interfaces. Thus, the techniques and systems describedherein for providing user interfaces that enable a user to configuresource type definitions are applicable to both on-premises andcloud-based service contexts, or some combination thereof.

3.0. Source Type Management

As indicated above, source type definitions may relate to severalaspects of how a data intake and query system collects, indexes, stores,and performs searches on data in both on-premises and cloud-basedservice environments. Thus, the ability for users to create andcustomize source type definitions can greatly facilitate the ability forusers to load virtually any type of data into a data intake and querysystem for analysis.

Furthermore, the use of source type definitions by a data intake andquery system to interact with data consumed by the system may occuracross multiple system components. For example, one or more propertiesof a source type definition may be referenced by an indexer duringparsing and indexing of data received. One or more other properties ofthe source type definition may be referenced by a search head duringprocessing of a search query. Thus, to ensure that data of a particularsource type is handled consistently at different processing phaseswithin the system, multiple system components may access the same sourcetype definition and any updates to a source type definition.

According to various embodiments described herein, a data intake andquery system provides one or more graphical user interfaces that enableusers to create, modify, and delete source type configurations.Furthermore, techniques are described for generating or modifying one ormore configuration files based on input received via the one or moregraphical user interfaces, and synchronizing these configuration filesacross various system components of the data intake and query system.The ability for users to configure source type definitions usingintuitive graphical user interfaces may, among other benefits, alleviateadministrators of a data intake and query system from many otherwisemanual and time-consuming maintenance steps.

Furthermore, by enabling synchronization of source type definitionsacross components of a data intake and query system, end users may bealleviated from understanding some of the more complex aspects of thesystem as it relates to source types (e.g., understanding what anindexer does separately from a search head, understanding the roles of acluster master and a deployment server, etc.). This abstraction of thesystem for end users may be beneficial both to users without directaccess to the individual computing resources implementing one or more ofthe system components and those who may not desire to undertake learninghow the individual system components operate separately from the systemas a whole.

3.1. Configuring Source Types

FIG. 9 is a flow diagram illustrating an example process for providinggraphical user interfaces for configuring source types, storing a sourcetype definition in one or more configuration files based on user input,and synchronizing source type definitions across various components of adata intake and query system. The various elements of flow 900 may beperformed in a variety of systems, including systems such as system 100and system 800 described above. In an embodiment, each of the processesdescribed in connection with the functional blocks described below maybe implemented using one or more computer programs, other softwareelements, and/or digital logic in any of a general-purpose computer or aspecial-purpose computer, while performing data retrieval,transformation, and storage operations that involve interacting with andtransforming the physical state of memory of the computer.

At block 902, a system component causes display of a graphical userinterface comprising a list of source type definitions which currentlyexist in the system. For example, one or more components of a system 100(e.g., a forwarder, indexer, or search head) or a system 800 maycomprise a web server configured to provide browser-based interfacesthat generally enable users to interact with the system. A user may, forexample, navigate a web browser to a particular URL to access the systeminterfaces. In another embodiment, other interfaces may be used such asa standalone application executing on a system component or clientdevice that interfaces with the system component. In general, one ormore of the graphical user interfaces may comprise interfaces forconfiguring one or more aspects of the system 100. For example, theinterfaces may provide interface components that enable a user toconfigure system level settings, settings related to data input sources,settings related report generation, etc.

In one embodiment, one or more of the graphical user interfacesgenerated by a system component include interfaces for configuringsource type definitions. For example, a user may use a web browser orother application to select a link from a system menu to configuresource type definitions. In response to receiving the user selection, acomponent of the system may cause display of one or more graphical userinterfaces that includes a list of existing source type definitions andfurther enable a user to select particular source type definitions formodification or deletion, or to select other interface components tocreate a new source type definition.

FIG. 10 depicts an example graphical user interface which lists existingsource type definitions of a data intake and query system. Interface1000, for example, comprises a menu bar 1002, and a table 1004 whichlists a number of existing source type definitions. Menu bar 1002includes menu options that enable a user to navigate to variousinterface screens provided by the system. For example, using menu bar1002, a user may navigate to interfaces for configuring various aspectsof the system, to perform searches on data stored in the system, or tointeract with the system in other ways. Interface 1000, for example, mayhave been generated in response to a user selecting an option from the“Settings” menu item in menu bar 1002 corresponding to “Source Types”indicating a desire to configure system source types.

Table 1004 displays several rows listing existing source typedefinitions including additional fields providing information related toeach listed source type definition. For example, the first row in table1004 corresponds to a source type definition labeled “log 4j,” thesecond row corresponds to a source type definition labeled“cisco_syslog,” and so forth. A system component generating interface1000 may obtain the data displayed in table 1002, for example, byexamining one or more configuration files, extracting items ofinformation for each source type definition included in theconfiguration files, and generating a table listing the extractedinformation items. Each row of table 1004 may include one or morehyperlinks that link to a separate graphical user interface forconfiguring particular source type definitions. For example, each labelin the “Name” column may operate as a hyperlink to a separate interfacefor configuring the source type definition corresponding to the selectedlabel. A “Status” column may indicate for each of the listed source typedefinitions whether the definition is currently enabled or disabled foruse by the system. An “Action” column may include, for example, one ormore hyperlinks or other interface components that link to separateinterfaces to modify or delete a particular source type definition.Interface 1000 may further include one or more interface components forcreating a new source type definition, or deleting an existingdefinition.

Referring again to FIG. 9, at block 906, in response to receiving aselection of a particular source type definition, or in response toreceiving a selection to create a new source type definition, a systemcomponent causes display of a graphical user interface for configuring asource type definition. In an embodiment, a graphical user interface forconfiguring a source type definition generally may include one or moreinterface components which enable a user to specify values associatedwith one or more attributes of a source type definition. For example,one interface component may relate to attributes that define how thesystem segments data of the corresponding source type into events,another interface component may relate to attributes for how the systemgenerates timestamps for each generated event, and so forth. If the userselected an existing source type definition for modification at block904, one or more of interface components may be pre-populated withexisting configuration information for the selected source type.

FIG. 11 depicts a portion of an example graphical user interfacedisplaying one or more interface components for configuring a sourcetype definition. Region 1102, for example, comprises interfacecomponents that enable a user to specify a rule for determining eventboundaries in input data associated with the corresponding source type.By selecting one of the buttons in region 1102, for example, a user mayindicate for the source type definition under modification that theboundaries of events are to be automatically determined by the system,determined based on each line included in input data, or determinedbased on a customizable regular expression that defines eventboundaries.

Region 1104 comprises interface components that enable a user to specifya rule for determining a timestamp to be associated with each eventassociated with the source type. For example, a user may select a buttonin region 1104 to indicate that timestamps are to be automaticallygenerated for each event, based on a time corresponding to when eachevent is created, or based on other timestamp rules such as a regularexpression that identifies where a time value for each event is storedwithin the events. A “Timestamp format” may enable a user to specify astring representing a time stamp format to use when searching fortimestamps within input data.

Region 1106 includes one or more “greenfield” interface components thatenable a user to specify values for any attribute-value pair supportedby the system. For example, a user may review documentation for the dataintake and query system and determine that an attribute named “CHARSET”may be configured to identify a character encoding associated with asource type. As depicted in the example of region 1106, a user mayprovide input specifying the name of the attribute (e.g., CHARSET in the“Name” field) and further specifying a value of “UTF-8,” correspondingto the UTF-8 character encoding. The rows below the first row includeother specified attribute-value pairs, including values for a“NO_BINARY_CHECK” and a “disabled” attribute.

Region 1108 comprises one or more interface components that enable auser to specify other delimited settings, including a field delimitersetting, a quote character setting, a file preamble setting, and one ormore field name settings. Each of these settings may enable a user toadjust other settings related to how input data of the source type isparsed. For example, a “Field delimiter” setting may enable a user tospecify a character that identifies how to separate input data intoseparate field; a “Quote character” setting may enable a user to specifya character that identifies when particular text within input data iscontained within quotes; a “Field preamble” setting may enable a user toprovide input specifying a regular expression or other instructions toidentify one or more lines at the beginning of input data containing apreamble section that is to be ignored during parsing; a “Field names”setting may enable a user to specify one or more regular expressions orother rules for determining a field name for fields extracted from inputdata.

Referring again to FIG. 9, at block 908, user input is received by asystem component, the input comprising one or values corresponding tovarious attributes of a source type definition. For example, a userproviding the input to one or more graphical user interfaces may selecta button (e.g., a “save” button) or other interface component to savethe configuration input and, in response, a web browser or otherapplication used by the user to provide the input sends one or morevalues corresponding to the input to a system component. The inputgenerally may include one or more values corresponding to one or moreattributes of the source type definition including, for example, one ormore numeric values, text strings, regular expressions, etc. In anembodiment, the input may be received as it is provided by the user andwithout an express action by the user to save the input.

At block 910, a source type definition under configuration optionallymay be previewed. In general, a preview of a source type definitiongenerally may include any display that indicates how one or moreattributes of the configured source type definition are applied to inputdata associated with the source type. For example, the display mayinclude a display that shows a sample of data parsed into a plurality ofevents based on the specified event boundary rules. The display may alsoinclude an indication of a timestamp to be applied to each event, andthe contents of one or more fields extracted from each event. Byreviewing the contents of the parsed events, timestamps, and otherattributes, a user can determine whether the configured source typedefinition results in a parsing of input data as the user expected. Ifthe user determines that the sample events are not correctly parsed, theuser may return to the configuration interface for the source typedefinition and further modify the definition until a desired result isreached.

FIG. 12 depicts an example graphical user interface which includes apreview displaying sample input data parsed according to one or moreattributes of a source type definition. Interface 1200, for example,includes a region 1202 comprising a table listing several example eventsderived from input data based on a configured source type definition.The left side of interface 1200 depicts collapsed interface componentgroups for configuring a source type definition, for example, similar tothose depicted in FIG. 11. For example, a user may configure a sourcetype definition using one or more of the interface components depictedin FIG. 11, and subsequently a preview of example events derived frominput data based on the configured source type definition may bedisplayed. The preview displayed in region 1202 may be displayed inresponse to a user selecting an interface component indicating that theuser has completed configuring a source type definition, the preview maybe displayed in real-time as the user configures the source typedefinition, etc.

Each of the rows of the table in region 1202, for example, illustrate anindividual event parsed from sample input data according to the sourcetype definition under configuration. Region 1202 provides otherindications of how a source type definition is being applied, forexample, by highlighting portions of each event indicating data fromwhich a timestamp is derived for each event. Other example interfacesmay provide other preview information, including an indication of fieldsparsed from events or transformations applied to data contained withinthe events.

3.2. Synchronizing Source Type Configurations

In one embodiment, a user may provide input to save a source typedefinition under configuration. In response to receiving the input tosave a configured source type definition, one or more attribute-valuepairs and any other configuration information related to the source typedefinition as configured by the user are saved in one or moreconfiguration files. The configuration files generally may be stored instorage local to the component of the data intake and query system whichgenerated the interfaces used to create or modify the source typedefinition. For example, if a user accesses an interface generated by anindexer of system 100 to configure a source type definition, theresulting source type definition may be saved in one or moreconfiguration files stored at that indexer. However, source typedefinitions generally include configuration data that is relevant tomore than one component of a data intake and query system (e.g., otherindexers, forwarders, search heads, etc.). Thus, in one embodiment,source type definitions are synchronized across multiple systemcomponents to ensure that data of a particular source type isconsistently processed throughout a data intake and query system.

Referring again to FIG. 9, at block 914, a source type definitioncreated or modified by a user at one system component is synchronizedwith other system components. In general, synchronizing a source typedefinition may include sending one or more configuration filescontaining the newly created or modified source type definition to oneor more other system components via a network. Sending the one or moreconfiguration files may include sending the entirety of the files, oronly a portion of the configuration files which contain the newlycreated or modified source type definition may be synchronized. Each ofthe system components receiving the updated configuration files orportions thereof may store the updated configure files locally to thatcomponent for subsequent use.

In one embodiment, a process for synchronizing one or more configurationfiles containing source type definitions may involve separatesynchronization steps for different types of system components. Forexample, as illustrated in reference FIG. 1 and FIG. 8, a data intakeand query system generally may include a number of system components ofvarious types, including one or more forwarders, indexers, and searchheads. In one embodiment, synchronizing the one or more configurationfiles may include sending the one or more configuration files toparticular system components that are responsible for managing othersystem components. These particular system components may in turnsynchronize the configuration files with other components of aparticular type.

For example, a data intake and query system may include a plurality ofindexers, referred to as an indexer cluster, which are managed by asystem component referred to as a “master node.” In general, a masternode manages the operation of an indexer cluster, including maintaininga list of peer indexers currently in operation and managing replicationof data across peer indexers. In an embodiment, synchronizing one ormore configuration files with an indexer cluster may include sending theconfiguration files to the master node, which is then configured tosynchronize the configuration files across indexers. The master nodemay, for example, maintain a list of IP addresses or other connectionidentifiers associated with each indexer in the cluster that enables themaster node to send the configuration files to the indexers over anetwork. Each indexer in the cluster may receive the configuration filesfrom the master node and store the files local to the indexer forsubsequent use.

Similarly, synchronizing one or more configuration files may includesending the configuration files to a system component responsible formanaging operation of one or more search heads, referred to as a “searchcluster master.” A search cluster master generally is responsible formanaging a cluster of search heads, and may be configured to distributeone or more updated configuration files to each of the search heads forwhich it is responsible. Techniques for distributing configurationinformation among search heads are described in U.S. patent applicationSer. No. 14/448,919, entitled “CONFIGURATION REPLICATION IN A SEARCHHEAD CLUSTER”, filed on 31 Jul. 31 2014, and which is herebyincorporated by reference in its entirety for all purposes.

In one embodiment, synchronization of configuration files with one ormore forwarders may comprise sending the configuration files to a systemcomponent referred to as a “deployment server.” In general, a deploymentserver may be configured to distribute configurations and contentupdates to various components of a data intake and query system,including forwarders. In one embodiment, each forwarder 104 may beconfigured to periodically communicate with a particular deploymentserver to check for any configuration updates. If the deployment serverindicates that one or more configuration files updates are available(e.g., one or more configuration files containing new or modified sourcetype definitions), the deployment server may push to the forwarder orthe forwarder may download the configuration file updates from thedeployment server. The forwarder may then save the one or moreconfiguration files to local storage.

In an embodiment, an apparatus comprises a processor and is configuredto perform any of the foregoing methods.

In an embodiment, a non-transitory computer readable storage medium,storing software instructions, which when executed by one or moreprocessors cause performance of any of the foregoing methods.

Note that, although separate embodiments are discussed herein, anycombination of embodiments and/or partial embodiments discussed hereinmay be combined to form further embodiments.

4.0. Implementation Mechanisms—Hardware Overview

According to an embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 13 is a block diagram that illustrates a computersystem 1300 upon which an embodiment may be implemented. Computer system1300 includes a bus 1302 or other communication mechanism forcommunicating information, and a hardware processor 1304 coupled withbus 1302 for processing information. Hardware processor 1304 may be, forexample, a general purpose microprocessor.

Computer system 1300 also includes a main memory 1306, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 1302for storing information and instructions to be executed by processor1304. Main memory 1306 also may be used for storing temporary variablesor other intermediate information during execution of instructions to beexecuted by processor 1304. Such instructions, when stored innon-transitory storage media accessible to processor 1304, rendercomputer system 1300 into a special-purpose machine that is customizedto perform the operations specified in the instructions.

Computer system 1300 further includes a read only memory (ROM) 1308 orother static storage device coupled to bus 1302 for storing staticinformation and instructions for processor 1304. A storage device 1310,such as a magnetic disk, optical disk, or solid-state drive is providedand coupled to bus 1302 for storing information and instructions.

Computer system 1300 may be coupled via bus 1302 to a display 1312, suchas a cathode ray tube (CRT), for displaying information to a computeruser. An input device 1314, including alphanumeric and other keys, iscoupled to bus 1302 for communicating information and command selectionsto processor 1304. Another type of user input device is cursor control1316, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1304 and for controlling cursor movement on display 1312. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 1300 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1300 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1300 in response to processor 1304 executing one or moresequences of one or more instructions contained in main memory 1306.Such instructions may be read into main memory 1306 from another storagemedium, such as storage device 1310. Execution of the sequences ofinstructions contained in main memory 1306 causes processor 1304 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical disks, magnetic disks, or solid-state drives, suchas storage device 1310. Volatile media includes dynamic memory, such asmain memory 1306. Common forms of storage media include, for example, afloppy disk, a flexible disk, hard disk, solid-state drive, magnetictape, or any other magnetic data storage medium, a CD-ROM, any otheroptical data storage medium, any physical medium with patterns of holes,a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip orcartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 1302. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1304 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 1300 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 1302. Bus 1302 carries the data tomain memory 1306, from which processor 1304 retrieves and executes theinstructions. The instructions received by main memory 1306 mayoptionally be stored on storage device 1310 either before or afterexecution by processor 1304.

Computer system 1300 also includes a communication interface 1318coupled to bus 1302. Communication interface 1318 provides a two-waydata communication coupling to a network link 1320 that is connected toa local network 1322. For example, communication interface 1318 may bean integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example,communication interface 1318 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, communicationinterface 1318 sends and receives electrical, electromagnetic or opticalsignals that carry digital data streams representing various types ofinformation.

Network link 1320 typically provides data communication through one ormore networks to other data devices. For example, network link 1320 mayprovide a connection through local network 1322 to a host computer 1324or to data equipment operated by an Internet Service Provider (ISP)1326. ISP 1326 in turn provides data communication services through theworld wide packet data communication network now commonly referred to asthe “Internet” 1328. Local network 1322 and Internet 1328 both useelectrical, electromagnetic or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1320 and through communication interface 1318, which carrythe digital data to and from computer system 1300, are example forms oftransmission media.

Computer system 1300 can send messages and receive data, includingprogram code, through the network(s), network link 1320 andcommunication interface 1318. In the Internet example, a server 1330might transmit a requested code for an application program throughInternet 1328, ISP 1326, local network 1322 and communication interface1318.

The received code may be executed by processor 1304 as it is received,and/or stored in storage device 1310, or other non-volatile storage forlater execution.

5.0. Example Embodiments

In an embodiment, a method or non-transitory computer readable mediumcomprises: causing, by a system component of a data intake and querysystem comprising a plurality of system components, display of agraphical user interface for configuring a source type definitioncorresponding to a particular source type, the source type definitionincluding one or more attributes related to parsing input data of thesource type to generate one or more events; receiving, via the graphicaluser interface, user input corresponding to values for the one or moreattributes; storing, in one or more configuration files, the one or morevalues in association with the one or more attributes; causing the oneor more configuration files to be stored on at least two systemcomponents of the plurality of system components.

In an embodiment, the method or computer readable medium furthercomprises: subsequent to causing the one or more configuration files tobe stored on at least two system components of the plurality of systemcomponents, receiving, by a particular system component of the pluralityof system components, particular input data associated with theparticular source type; generating a plurality of events from theparticular input data based on one or more of the one or more valuesstored in the one or more configuration files.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: subsequent causing the one or more configurationfiles to be stored on at least two system components of the plurality ofsystem components, receiving, by a search head, a query for data of thesource type; processing, by the search head, the query based on one ormore of the one or more values stored in the one or more configurationfiles.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: wherein the source type definition further includesone or more attributes related to determining a timestamp for each eventof the one or more events.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: wherein the source type definition further includesone or more attributes related to extracting one or more fields fromeach event of the one or more events.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: wherein the one or more one or more values stored inassociation with the one or more attributes comprises a regularexpression.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: wherein causing the one or more configuration filesto be stored on at least two of the two or more system componentsincludes sending the one or more configuration files to one or more ofthe at least two of the two or more system components.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: wherein causing the one or more configuration filesto be stored on at least two system components of the plurality ofsystem components is performed automatically in response to userproviding input to save the source type definition.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: causing display of a second graphical user interfacedisplaying a list of existing source type definitions; receiving aselection of a particular source type definition from the list ofexisting source type definitions; wherein the graphical user interfaceis caused to be displayed in response to receiving the selection of theparticular source type definition.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: causing display of a second graphical user interfacedisplaying a list of existing source type definitions; wherein thesecond graphical user interface includes one or more interfacecomponents to create a new source type definition or to delete anexisting source type definition.

In an embodiment, the method or non-transitory computer readable mediumfurther comprises: causing display of a preview displaying particularinput data parsed according to one or more particular attributes of thesource type definition.

In an embodiment, a method or non-transitory computer readable mediumcomprises: wherein the plurality of system components includes one ormore of an indexer, a forwarder, and a search head.

In an embodiment, a method or non-transitory computer readable mediumcomprises: wherein the input data is one or more of a file, a directoryof files, and a network event.

In an embodiment, a method or non-transitory computer readable mediumcomprises: wherein causing the one or more configuration files to bestored on at least two of the two or more system components includessending the one or more configuration files to a master node configuredto manage a cluster of indexers.

In an embodiment, a method or non-transitory computer readable mediumcomprises: wherein causing the one or more configuration files to bestored on at least two of the two or more system components includessending the one or more configuration files to a search cluster masterconfigured to manage a cluster of search heads.

In an embodiment, a method or non-transitory computer readable mediumcomprises: wherein causing the one or more configuration files to bestored on at least two of the two or more system components includessending the one or more configuration files to a deployment serverconfigured to manage one or more forwarders.

In an embodiment, a method or non-transitory computer readable mediumcomprises: receiving, by a particular system component of the pluralityof system components, particular input data associated with theparticular source type; based on one or more of the plurality of valuesstored in the one or more configuration files: generating a plurality ofevents from the particular input data; and determining a timestamp foreach event of the plurality of events.

In an embodiment, a method or non-transitory computer readable mediumcomprises: causing display of a preview displaying one or more eventsparsed from particular input data according to one or more particularattributes of the source type definition, and further providing anindication of one or more fields parsed from the one or more events.

6.0. Extensions and Alternatives

In the foregoing specification, embodiments have been described withreference to numerous specific details that may vary from implementationto implementation. The specification and drawings are, accordingly, tobe regarded in an illustrative rather than a restrictive sense. The soleand exclusive indicator of the scope of the embodiments, and what isintended by the applicants to be the scope of the embodiments, is theliteral and equivalent scope of the set of claims that issue from thisapplication, in the specific form in which such claims issue, includingany subsequent correction.

In drawings, various system components are depicted as beingcommunicatively coupled to various other components by arrows. Thesearrows illustrate only certain examples of information flows between thecomponents of the depicted systems. Neither the direction of the arrowsnor the lack of arrow lines between certain components should beinterpreted as indicating the absence of communication between thecertain components. Indeed, each component of the depicted systems mayfeature an open port, API, or other suitable communication interface bywhich the component may become communicatively coupled to othercomponents of the depicted systems as needed to accomplish any of thefunctions of the systems described herein.

The invention claimed is:
 1. A method, comprising: causing, by a dataintake and query system comprising a plurality of system components,display of a graphical user interface for configuring a source typedefinition corresponding to a particular source type of an informationtechnology environment, the source type definition including a pluralityof attribute-value pairs to enable the data intake and query system togenerate one or more events from data of the particular source typegenerated by at least one component in the information technologyenvironment; wherein the graphical user interface includes one or moreattributes corresponding to one or more attribute-value pairs of theplurality of attribute-value pairs, and a user editable field forentering a value associated with at least one of the one or moreattributes; receiving, via the graphical user interface, user inputcorresponding to the value for the at least one of the one or moreattributes; storing, in one or more configuration files, the value inassociation with the at least one of the one or more attributes as oneof the one or more attribute-value pairs; and causing the one or moreconfiguration files to be stored on one or more system components of theplurality of system components.
 2. The method of claim 1, furthercomprising: subsequent to causing the one or more configuration files tobe stored on one or more system components of the plurality of systemcomponents, receiving, by a particular system component of the pluralityof system components, particular data of the particular source type; andgenerating a plurality of events from the particular data based on thevalue stored in the one or more configuration files.
 3. The method ofclaim 1, further comprising: subsequent to causing the one or moreconfiguration files to be stored on one or more system components of theplurality of system components, receiving, by a search head, a query fordata of the particular source type; and processing, by the search head,the query based at least in part on the value stored in the one or moreconfiguration files.
 4. The method of claim 1, wherein the source typedefinition further includes one or more attributes related todetermining a timestamp for each event of the one or more events.
 5. Themethod of claim 1, wherein the source type definition further includesone or more attributes related to extracting one or more fields from thedata of the particular source type for each event of the one or moreevents.
 6. The method of claim 1, wherein the value stored inassociation with the at least one of the one or more attributescomprises a regular expression.
 7. The method of claim 1, whereincausing the one or more configuration files to be stored on one or moresystem components includes sending the one or more configuration filesto two or more system components.
 8. The method of claim 1, whereincausing the one or more configuration files to be stored on one or moresystem components of the plurality of system components is performedautomatically in response to a user providing input to save the sourcetype definition.
 9. The method of claim 1, wherein the graphical userinterface is a first graphical user interface, the method furthercomprising: causing display of a second graphical user interfacedisplaying a list of existing source type definitions; and receiving aselection of a particular source type definition from the list ofexisting source type definitions, wherein the first graphical userinterface is caused to be displayed in response to receiving theselection of the particular source type definition.
 10. The method ofclaim 1, wherein the graphical user interface is a first graphical userinterface, the method further comprising: causing display of a secondgraphical user interface displaying a list of existing source typedefinitions, wherein the second graphical user interface includes one ormore interface components to create a new source type definition or todelete an existing source type definition.
 11. The method of claim 1,further comprising causing display of a preview displaying particulardata of the particular source type parsed according to one or moreparticular attributes of the source type definition.
 12. The method ofclaim 1, wherein the plurality of system components includes one or moreof an indexer, a forwarder, and a search head.
 13. The method of claim1, wherein the data of the particular source type is one or more of afile, a directory of files, and a network event.
 14. The method of claim1, wherein causing the one or more configuration files to be stored onone or more system components includes sending the one or moreconfiguration files to a master node configured to manage a cluster ofindexers.
 15. The method of claim 1, wherein causing the one or moreconfiguration files to be stored on one or more system componentsincludes sending the one or more configuration files to a search clustermaster configured to manage a cluster of search heads.
 16. The method ofclaim 1, wherein causing the one or more configuration files to bestored on one or more system components includes sending the one or moreconfiguration files to a deployment server configured to manage one ormore forwarders.
 17. The method of claim 1, further comprising:receiving, by a particular system component of the plurality of systemcomponents, particular data of the particular source type; based atleast in part on the value stored in the one or more configurationfiles: generating a plurality of events from the particular data; anddetermining a timestamp for each event of the plurality of events. 18.The method of claim 1, further comprising causing display of a previewdisplaying one or more events generated from particular data of theparticular source type according to one or more particular attributes ofthe source type definition, and further displaying an indication of oneor more fields parsed from the particular data.
 19. Non-transitory,computer-readable storage media storing computer-executable instructionsthat, when executed by one or more computing devices of a data intakeand query system, cause the one or more computing devices to: causedisplay of a graphical user interface for configuring a source typedefinition corresponding to a particular source type of an informationtechnology environment, the source type definition including a pluralityof attribute-value pairs to enable the data intake and query system togenerate one or more events from data of the particular source type,wherein the data of the particular source type is generated by at leastone component in the information technology environment; receive, viathe graphical user interface, user input for a value associated with atleast one attribute of the data of the particular source type, the atleast one attribute corresponding to at least one attribute-value pairof the plurality of attribute-value pairs; store, in one or moreconfiguration files, the value in association with the at least oneattribute as one of the plurality of attribute-value pairs; and causethe one or more configuration files to be stored on one or more systemcomponents of the data intake and query system.
 20. The non-transitory,computer-readable storage media of claim 19, wherein thecomputer-executable instructions, when executed by the one or morecomputing devices, cause the system component to: subsequent to causingthe one or more configuration files to be stored on one or more systemcomponents of the data intake and query system, receive, by a particularsystem component of the data intake and query system, particular dataassociated with the particular source type; and generate a plurality ofevents from the particular data based at least in part on the valuestored in the one or more configuration files.
 21. The non-transitory,computer-readable storage media of claim 19, wherein thecomputer-executable instructions which, when executed by the one or morecomputing devices, cause the one or more computing devices to:subsequent to causing the one or more configuration files to be storedon one or more system components of the plurality of system components,receive, by a search head, a query for data of the particular sourcetype; and process, by the search head, the query based at least in parton the value stored in the one or more configuration files.
 22. Thenon-transitory, computer-readable storage media of claim 19, wherein thesource type definition further includes one or more attributes relatedto determining a timestamp for each event of the one or more events. 23.The non-transitory, computer-readable storage media of claim 19, whereinthe source type definition further includes one or more attributesrelated to extracting one or more fields from the data of the particularsource type for the one or more events.
 24. The non-transitory,computer-readable storage media of claim 19, wherein the value stored inassociation with the at least one attribute comprises a regularexpression.
 25. The non-transitory, computer-readable storage media ofclaim 19, wherein to cause the one or more configuration files to bestored on one or more system components, the computer-executableinstructions when executed, cause the one or more computing devices tosend the one or more configuration files to two or more systemcomponents.
 26. The non-transitory, computer-readable storage media ofclaim 19, wherein the computer-executable instructions, when executed,automatically cause the one or more computing devices to cause the oneor more configuration files to be stored on the one or more systemcomponents of the data intake and query system in response to a userproviding input to save the source type definition.
 27. Thenon-transitory, computer-readable storage media of claim 19, wherein thegraphical user interface is a first graphical user interface, andwherein the computer-executable instructions, when executed by the oneor more computing devices, further cause the one or more computingdevices to: cause display of a second graphical user interfacedisplaying a list of existing source type definitions; and receive aselection of a particular source type definition from the list ofexisting source type definitions, wherein the first graphical userinterface is caused to be displayed in response to receiving theselection of the particular source type definition.
 28. Thenon-transitory, computer-readable storage media of claim 19, wherein thecomputer-executable instructions, when executed by the one or morecomputing devices, cause display of a preview displaying particular rawmachine data parsed according to one or more particular attributes ofthe source type definition.
 29. The non-transitory, computer-readablestorage media of claim 19, wherein the data intake and query systemincludes at least one of an indexer, a forwarder, or a search head. 30.A system, comprising: one or more computing devices of a data intake andquery system, the data intake and query system comprising a plurality ofsystem components; and non-transitory, computer-readable storage mediain communication with the one or more computing devices and storing alocal copy of one or more configuration files, wherein the one or morecomputing devices are configured to: cause display of a graphical userinterface for configuring a source type definition corresponding to aparticular source type of an information technology environment, thesource type definition including a plurality of attribute-value pairs toenable the data intake and query system to generate one or more eventsfrom data of the particular source type generated by at least onecomponent in the information technology environment; receive, via thegraphical user interface, user input for a value associated with atleast one attribute of the data of the particular source type, the atleast one attribute corresponding to at least one attribute-value pairof the plurality of attribute-value pairs; store in the one or moreconfiguration files, the value in association with the at least oneattribute as one of the plurality of attribute-value pairs; and causethe one or more configuration files to be stored on one or more systemcomponents of the data intake and query system.